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ABSTRACT 

Objectives and progress of the Student Impacts and 
Outcomes Committee created in the summer of 1985 at Eastern New 
Mexico University are reported. The committee was formed to measure 
and assess the impact that 4 years of college have on a student, and 
intends to track an entire entering freshman class from the time the 
students apply until they leave, with a follow-up 2 to 5 years later. 
It will then track every fourth entering freshman class, i.e., as one 
class enters its senior year, the new entering freshman class will 
become the study population. Five subgroups were formed to consider 
knowledge outcomes, skills outcomes, attitudes and values, 
relationship with the university, and occupational outcomes. Each 
subgroups will identify the most appropriate data to collect and 
possible data collection instruments. Updates of the progress of each 
of the five subgroups are presented. Two recommendations are offered: 

(1) to measure general knowledge outcomes, the short form of the 
American College Testing Program's College Outcome Measures Program 
Examination (ACT COMP) should be used for pre- and post-testing; and 

(2) the ACT Entering Student Survey, the ACT 

Withdrawing/Non-returning Student Survey and the ACT Student Opinion 
Survey should also be employed. Two policy recommendations are also 
offered: the mission statement should be rewritten, and the 
philosophy of the general education component is needed in writing. 
(SW) 
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Prefac2 

America's belief in the value of a college education is diminishing. For a 
number of years, postsecondary education was viewed as the stepping stone to 
success , in the market place and in lif e . The federal government supported 
postsecondary education with contracts and grants, and awarded financial aid to 
students. State government 3 built new campuses, increased programs, and developed 
new funding sources. The commitment to postsecondary education was pervasive. 

However, over the past few years postsecondary education's support has crumbled. 
From every quarter, critics challenge colleges and universities to demonstrate that 
they are efficient and effective. Declining enrollments have forced institutions 
into a mad scramble for students and funds. States are finding themselves with too 
many irsti tut ions and too little revenue to continue the level of support of past 
years. As agendas change, and as America grows grayer, postsecondary education must 
struggle to maintain present levels of support much less increase that support to 
pay for new programs, for increased salaries, and to repair Deteriorating physical 
plants. 

Certainly the erosion of support is evident here in New Mexico. After years of 
building new junior colleges and branch campuses, the state of New Mexico is now 
experiencing declining revenues, as extractive industries in particular face reduced 
demand and increased foreign competition. Additionally, other constituencies within 
the state are seeking larger shares of the shrinking state dollar. Initiatives have 
already been proposed to reduce the size and scope of postsecondary education in New 
Mexico. As revenues continue to be smaller tnan anticipated, reductions and 
eliminations of both programs and possibly entire campuses are possible. 
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Not unsurprisingly, colleges and universities have begun to react to these 
life-threatening assaults. For the first time in memory, colleges and universities 
are having to justify their existence, their value, to a world that no longer 
assumes that a college education can provide both tangible and intangible rewards 
unattainable elsewhere in our society. Currently, a number of studies are underway 
at universities across the United States to measure and assess the impact they have 
on their students. 

Eastern New Mexico University has begun its own efforts to measure and assess the 
impact that four years of college have on a student. In the summer of 1985, 
President Robert Matheny created a Student Impacts and Outcomes Committee. Tk& 
charge to this committee was three fold: (1) to design and implement a comprehensive 
student outcomes and impact study; (2) to oversee the study and protect its 
integrity; and (3) based on the work of the committee, to make policy 
recommendations to the Vice President for Planning and Analysis. 
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Introduction 

This report is to inform the campus of the progress the Student Impacts and Outcomes 
Committee has made during the last several months. Many issues have been raised, 
several significant decisions made, and policy recommendations proposed. In order to 
present the material clearly, this report is divided into the following sections: 
(1) Background; (2) Subgroup Reports; (3) Recommendations for the Study; (4) Policy 
Recommendations; and (5) Summary and Conclusion?. 

Background 

In order to establish a better understanding of the "impact Eastern New Mexico 
University has on its students and to provide "hard" data upon which to base 
curricular decisions and develop policy, President Matheny commissioned the Student 
Impacts and Outcomes Committee. At the first meeting, the committee agreed that its 
major responsibilities were as follows: ".,.(!) to oversee the study in order to 
protect its integrity; (2) to identify what data to collect and when to collect it; 
(3) to issue reports; (,4) to authorize specific analysis and reports; and (5) to 
make policy recommendations to the Vice President for Planning and Analysis based 
upon findings reported." 

The Office of Institutional Research provides staff support to the committee. 
This support includes but is not limited to providing technical assistance in the 
development of the <*tudy, developing data management and analysis tools, drafting 
reports, and making recommendations to the committee on needed actions, possible 
analysis and reports. 

A number of commitments have guided the work of the committee so far. First , the 
committee recognizes that social science research often has methodological problems 
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which cast doubt on both the validity and reliability of a study. Clearly a study 
of this magnitude, conducted with limited human and financial resources, will 
inevitably carry with it serious validity and reliability concerns. While the study 
results may be questioned, the committee believes that it is better to note the 
problems and potential intervening variables that may contaminate the study than 
spend years refining the study in hopes of discovering the perfect design. 

Second, the committee believes that tentative judgments are better than no 
judgments at all. Therefore, the committee will try to identify factors that it 
thinks are important, while noting the conceptual and methodological issues that 
might produce in different conclusions- The requirements of scholarship require 
that the f Mnmittee openly discuss the possibility of error, yet professional 
obligation also requires the committee to make judgments and form conclusions. 

Third, the committee wants to provide the broadest possible dissemination of 
information that it gathers to the university community. Departments and individual 
faculty members can probably make the best use of collected information. Therefore, 
the committee plans to be as open as possible about its efforts and invites 
questions about its work at any time. This report is part of the effort to explain 
the work of the committee to colleagues. 

Finally, the committee is committed to drawing conclusions and making 
recommendations whenever possible. This means that the committee will not be just a 
collector of data but also a user of data. The committee's task will not be to 
single out a group, an area, or a department for scrutiny. If, however, in the 
course of our work we discover areas of concern, we will address those concerns in 
reports and recommendations. Some of our conclusions will be applauded, some 
scorned. Yet the committee is committed to the principle of free inquiry that 
should and must characterize a university community. 

° ft 



The starting point for this study was the initial proposal prepared by Dr. Robert 
Wilkinson. The proposal outlined the general design and methodology, identified 
major decision points which this committee would need to address, and provided a 
visual design of the data collection points. 

As a brief overview, the study is designed as a comprehensive outcomes study to 
be implemented on a cyclical basis (every four years). The committee's intention is 
to track an entire entering freshman class from the time they apply to Eastern until 
they leave with £ follow-up two to five years later. Further, it is the intention of 
the committee to track every fourth entering freshman class. That is, as one class 
is entering its senior year, the new entering freshman class will become the new 
study population. This type of study incorporates both the traditional 
attrition/ retention design with the classical outcomes study, thereby providing a 
wealth of data about students. This type of research design will provide data that 
can be analyzed to show trends, changes, and between-class comparisons. Finally, 
since the committee is interested in studying a student's progression through the 
institution, data collection will start with the enrollment application process. 
However, for the purpose of the study, actual student contact will not begin until 
after the student has actually enrolled as an entering freshman. 

With its charge refined and with the initial proposal as a starting point, the 
committee used the following questions to guide its preliminary work. 

1. What is the committee measuring? 

2. What are the data sources? 

3. What procedures does the committee n^ed? 

4. What general considerations should guide the process? 
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It quickly became apparent that there was insufficient time to perfect the design, 
select appropriate data collection instruments, develop the accompanying data 
collection processes, ana then begin the study by the fall 1985 semester. Therefore, 
the committee elected to postpone initiation of the study for one year, while <* one 
year pilot study is conducted. The pilot study, already begun, will more adequately 
address the four central questions and prepare for the fall 1986 semester. Within 
the first year pilot study period, the committee set the following timetable to 
follow. 

December 1985 Draft proposal outlining the entire study will be 

prepared for presentation to the campus for review and 
comment • 

April 1986 Final guidelines for the study will be completed and a 

report prepared detailing all aspects of the study. 

May 1986 All necessary material are to be ordered and the first 

phase of the study initiated. 

The task before the committee is a monumental one, requiring a substantial 
commitment of time and energy by all involved. To provide a sense of direction, the 
committee was divided into five (5) subgroups based on a classification of possible 
outcomes: knowledge outcomes; skills outcomes; attitudes and values; relationship 
with the University; and occupational outcomes. Each subgroup is suppose to identify 
the most appropriate data tc collect and then identify possible data collection 
instruments. Each subgroup will make presentations to the committee. These reports 
are to include what they found, any limitations they may have identified, any 
problems they are having, and their recommendation to the committee. Finally as each 
subgroup works on its area, each will make recommendations to the committee on any 
policy issues that need to be advanced to the Vice President for Planning and 
Analysis. The subgroups are as follows. 
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1. Knowledge Outcomes 



Dr. George Mehaffy and Mr. T /id Noblett 



2. Skills Outcomes 



Dr. Thurman Elder and Dr. Pat Caldwell 



3. Attitudes and Values 



Dr. Eldon Walker and Dr. Renee Neely 



4. Relationship with University Dr. Richard Walsh and Ms. Carol Holden 



5. Occupational Outcomes 



Dr. Dale Davis and Mr. Winston Cox 



Data collection is a critical problem for each of the subgroups. First each 
subgroup must decide what data are necessary to form conclusions about student 
impacts and outcomes. If appropriate data are identified, can they be collected. 
Finally, the subgroup must identify data already collected, to avoid needless 
duplication of effort. 

Subgroup Reports 

Knowledge Outcomes 

The subgruup studying general knowledge has identified the ACT College Outcome 
Measures Program (COMP) Examination as the instrument that best measures what we 
want to measure. Furthermore, this subgroup recommends that the the ACT COMP 
Objective exam, a two-hour version of the longer exam, be used instead of the longer 
more complex exam. This will reduce both the financial and human cost involved in 
administering and scoring the exams. 

There are, however, tvo broad issues which still need to be addressed. Firi,t and 
foremost, should a pilot stucy be conducted to determine the utility of the results, 
test the examination process, and develop a baseline of information? Second, will 
the results of the COMP Exam be in a form which will iend itself to useful analysis 
and interpretat ion? 



ERIC 



11 



8 



Finally, there is the larger issue to this recommendation. Does .the ACT COMP Exam 
measure those things which we feel are important? Linked to this issue is another 
question: To what extent can Eastern claim credit for changes in its students? That 
is, are changes in students due to maturation, external variables, or derived from 
the experience of attending Eastern? 

Skills Outcomes 

The subgroups studying change in student's skills examined the ACT College 
Outcome Measure Program (Comp) and concurs with the Knowledge Outcomes subgroup that 
the short-form is a gcod choice for the study's purpose. Assessing computational 
skills requires an instrument that is modest in scope and in its degree of 
difficulty to be practical. Multiple-choice testing (the format most 
nationally-normed computational test employ) tends to ''test 11 discrete mathematical 
concepts or skills, rather than the ability to apply math knowledge and common sense 
to solve problems (e.g., reading a blueprint). This kind of testing, the subgroup 
feels, meets the purpose of this Committee'* ' tudy. We will continue, however, to 
examine other tests (from ACT or MAAi and consider generating a local exam. 

The short form COMP does not include any sort of writing on the part of the 
students, a skill that this subgroup feels must be Dart of this study. Not only 
does writing measure content mastery, it assesses that content on an individual 
basis, holistically and humanistically. Holistic scoring of student writing 
samples, by a cross-section of faculty, would be the best approach to this testing. 
Problems that arise are these: Given the bias possible in writing tasks, how are 
such tasks to be determined? How are the writing evaiuators to be selected and 
trained to score holistically? Since holistic scoring cannot represent absolute 
value but only meaning according, to established criteria and for a particular group, 
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can a writing sample stand as a meaningful measure of the university's educational 
experience for students? The subgroup believes that these problems can be solved 
and will continue to explore ways of implementing such a testing component. 

Attitudes and Values 

This subgroup has been meeting to determi. the best method of assessing the 
impact of a university experience on the values and attitudes of its students , 
Three major questions are being asked: (1) what values and attitudes are affected 
by the experience, (2) how can these best be measured, and (3) how can this be 
integrated into the larger study. 

The subcommittee is still attempting to reach agreement on the first question. 
Some attitudes toward self and society have been identified. However, the committee 
has not decided whether these are attitudes/values that can be (a) assumed to be 
changed by a college experience and necessary for inclusion or (b) operationally 
defined so they can be measured with any degree of reliability or validity. These 
decisions are complicated when any attempt is made to address onl, those values 
uniquely changed by the university and not by outside experiences. This question 
may be not answerable. If not, the lack of a control is a major limitation of the 
study. 

The members of the subcommittee are not ready to submit a proposal for the 
implementation of an attitudes/values portion of the study. The questions just 
presented have not been answered to :.he satisfaction of the subgroup. However, the 
subgroup has identified several variables that could be assessed through this study. 
These variables include intellectual development, personality changes (including 
such characteristics as maturity and responsibility), cultural development, moral 
development, and social development. These variables overlap and supplement the 
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attitudes/values toward self* personal goals, family, community, society, and others 
of primary interest to this study. 

One of the tasks before the subcommittee is narrowing the focus of the variables 
to be investigated. Preliminary discussions have centered around attempting to agree 
on those variables which (*) can be assumed to be influenced by a university 
experience, (bj can be measured reliably and validly, and {c) should be included as 
part of the outcomes study. So far, the subgroup has not reached any conclusions 
regarding those variables that meet these criteria. 

If an attempt is made to incorporate most of the areas of interest into a study 
of changes in attitudes/values, the subcommittee has several options. One 
alternative is to design an instrument. This would allow the committee to measure 
the changes in all or most of the categories at the same time. It would eliminate 
the need for sevexal different samples but would require the reliability and 
validity of the instrument be established before it could be used with any 
confidence. 

A second option is to select a few standardized instruments for use. This would 
eliminate the need to establish .he reliability and validity of the instruments but 
would mean additional sampling problems and some questions on interpretation of 
results because of differences in norms. 

The third and ideal alternative is to find one instrument that would cover as 
many of the areas of interest as possible, sampling and norm problems would be 
minimal. However, it might mean sacrificing some of the areas totally or finding 
alternative means of measuring the impact on them. This option is the preference of 
the subgroup. 

The subgroup is currently reviewing the relevant literature to determine how 
other institutions have measured changes in attitudes/values. Thus far, a single 
instrument that could be used to cover all of the areas has not been found. Some 
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methods are questionable in that they involve the administration of personality 
measures or pose ethical problems for the university. While the subgroup is not 
prepared to propose a specific method of data collection, it does suggest that a 
pilot effort in this area be conducted this spring followed by a follow-up study one 
year later. 

Relationship With the University 

The Relationship with the University subcommittee presented the following outline 
as a guide for the work they are doing. This outline proposes the types of questions 
to ask, the types of data that can be or are already collected, what similar types 
of activities are currently under way, and possible limitations to the study. 

I. GENERAL PLAN OF ACTION 
A. DEVELOP A PROFILE OF THE STUDENT IN THE UNIVERSITY. 

1. The profile will include such iten^ as: 

a. Academic 

b. Demographic 

c. Enrollment History 

d. Residency 

e. Residential Preference Patterns 

f. Extracurricular Plans 

g. Racial or Ethnic Background 

h. Religious Preference 

i. Past Student Profile 

j. Entering Students Interests 

k. - Graduating Seniors 

1. High School Background 

m. Ability Profile 

2. Sources of Data: 

a. ACT/Distance From Home 

b. ACT/First Year Predicted GPA f s 

c. ACT/SAT Scores 

d. Planning and Analysis 

e. Registrar 

f. Housing 
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g. Survey /ACT 

h. Survey /NCHEMS 

i. ACT/Home Community 

j. ACT/Extra Curricular Plans 

k. ACT/Family Income 

1. ACT/Profile 

m. Planning and Analysis, ENMU 

B. ESTABLISH INDIVIDUAL EDUCATIONAL DEVELOPMENT GOALS OF OUR 
STUDENTS 

1. Types of Questions: 

a. Intended Majors 

b. Purpose of Entering College 

c. Needs of Students 

d. Why Eastern was Selected 

2. Sources of Data: 

a. ACT, "Entering Student Survey" 

b. ACT, "Student Opinion Survey" 

c. ACT, Profile 

C. DETERMINE PLACEMENT PATTERNS IN THE UNIVERSITY SKILLS/ 
DEVELOPMENTAL STUDIES PROGRAM 

1. Areas: 

a. Math 

b. English 

c. Reading 

d. Intro, to University Studies 

2. Sources of Data: 

a. SAS - US/DS 

b. Departments: Math/English/ Reading/ACS 

D. RESEARCH STUDENT SUCCESS PATTERNS 

1. Factors: 

a. Retention/Attrition Rates 

b. Tracking Entering Students 

c. Opinions of Withdrawing/Non-Returning Students 

2. Sources of Data: 

a. SAS Retention and Tracking Records 

b. ACT/ "Withdrawing and Non-Returning Student Surveys" 

c. Planning and Analysis 
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E. REVIEW THE CHANGES THAT STUDENTS MAKE IN THEIR CHOICES OF MAJORS 

1. Sources of Data: 

a. College Records Clerks 

b. Advising Center 

c. Computer Center 

F. SiUDY THE BEHAVIOR PATTERNS OF ENROLLED STUDENTS 

1. Areas of Interest; 

a. Level of satisfaction/utilization with service 
programs* 

b. Level of satisfaction with academics 

c. Level of satisfaction with Admissions 

d. Level of satisfaction with rules and regulations 

e. Level of satisfaction with facilities 

f. Level of satisfaction with registration 

g. General level of satisfaction 

2« Source of Data: ACT - "Student Opinion Survey" 

G. STUDY THE LEVELS OF ACHIEVEMENT IN SUBSEQUENT EDUCATIONAL 
EXPERIENCES 

Source of Data: ACT, "Former Student Survey" 

H. ASSESS THE RELEVANCE AND CONTRIBUTION OF PAST EDUCATION TO 
SUBSEQUENT EDUCATIONAL EXPERIENCES 

Source of Data: ACT, "Former Student Survey" 



Occupational Outcomes 

After studying the materials provided by the Office of Institutional Research, it 
became apparent that there is no "one best way" to address the problem of measuring 
and assessing the impact Eastern has on its students and how alumni believe Eastern 
affected them. 

One of the major problems is that each institution tends to declare itself 
unique, and in so doing is saying, in effect, that to arrive at valid conclusions 
regarding the school, one ought to use a unique validating instrument. This in 
itself creates problems(e.g. , how can we make meaningful comparisons, to name but 
one). A more fundamental problem is that most educational institutions do a poor 
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job of framing their mission statements and the goals and objectives that are meant 
to make the mission statement a reality. 

Would the college of Liberal Arts be interested in using the same mission 
statement as the College of Business? In spite of the external pressures to be 
specific, we will be painting with a very broad brush even where we deal with 
meaningful and accurate goals and objectives. 

After presenting that caveat, this subcommittee submits the following 
recommendation. We have examined the "former-student questionnaire' 1 and "long-term 
alumni questionnaire" developed by NCHEMS and the "alumni survey" published by ACT. 
This subgroup recommends that the ACT "Alumni Survey" be used and that additional 
questions be developed and approved by the committee. 

Recommendation for the Study 

While the committee is still in the process of identifying the types of data to 
collect, what instruments to use, and what data collection procedures are needed, 
several definite recommendations have been advanced from the subgroups. Several of 
these recommendations were presented in the previous section but for the value of 
clarity they will be presented again along with several other recommendations which 
developed through general discussions. 

Recommendation 1. To measure general knowledge outcomes the "Knowledge Outcomes" 
subgroup recommends the short form of the M ACT COMP" exam in a 
pre- test/post-test setting, 

Recommendation 2. The "Relationship with the University" subgroup recommends the use 
of: (a) the ACT Entering Student Survey; (b) the ACT 
Withdrawing/Non-returning Student survey; and (c) the ACT Student 
Opinion Survey as the base data collection instrument. This 
subcommittee also recommends using the "student file" and the 
"transcript file" as secondary data sources for the study 
population. 
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Recommendation 3. The "Occupational Outcomes 11 subgroup recommends the use of the ACT 
Alumni survey plus thirty additional questions to collect data on 
the alumni. 

Recommendation 4. Members of the committee recommended that more thought 

and consideration be given to the possibilities of including 
transfer students in the study, as they seem to constitute a 
unique situation within the student body. 

Recommendation 5. The committee chair recommended the following time table be 
followed. 

December 1985 Draft proposal outlining the entire study 

will be prepared for presentation to the campus 
for review and comment. 

April 1986 Final guidelines for the study will be completed 

and a report prepared detailing all aspects of the 
study. 

May 1986 All necessary material are to be ordered and the 

first phase of the study initiated. 

Recommendation 6. The first recommendation tc come from the committee was to delay 
the initiation of the study for one year so there would be more 
time to develop the study and design the data collection 
procedure. 



Policy Recommendation 

Making policy recommendation to the Vice President for Planning and Analysis is 
one of the specific charges to this committee. As the committee has been designing 
the study, two specific issues seem to surface continually. The first issue centers 
around the mission of the university; the second issue focuses on the purpose of 
Eastern's general education component of the curriculum and the philosophical base 
upon which Eastern's curriculum is based. 

Recommendation 1. The mission statement should be rewritten and widely disseminated 

Recommendation 2. As part of the NCA Self -Study, there should be a written 

philosophy of Eastern's general education component. This written 



ERLC 



19 



16 

statement needs to identify what the purpose of Eastern's general 
education is and how that purpose is achieved* 

Summary and Conclusions 

The Student Impacts and Outcomes committee has begun addressing issues which are 
on the cutting edge of research in higher education. Undoubtedly the question of 
student outcome will continue to dominate the higher education agenda in the years 
to come. As this report demonstrates, the Student Impacts and Outcomes committee has 
made significant progress, but it still has a way to go before an Impacts and 
Outcomes Study can be initiated. The timetable the committee has set for itself is 
ambitious but realistic. The committee has been dealing with curricular issues and 
investigating policy areas in an effort to identify what types of data need to be 
collected to evaluate the "Eastern Experience. 11 This effort has already resulted in 
two policy recommendations, as well as encouraging discussion around the campus 
about the impact ENMU has on its students. 

So far, the following survey instruments have been recommended for use in the 
study: the ACT COMP Exam; the ACT Entering Student Survey; the ACT Withdrawing/ 
Nonreturning Student Survey; the ACT Alumni Survey; and the ACT Student Opinion 
Survey. There is still considerable work to be done in two areas: (1) determining 
how and if the committee should evaluate changes in attitudes and values and (2) how 
to evaluate computational and writing skills. 

As for the work vet to be done, there are several more stages before the process 
of implementing this study can begin. First, the committee has to agree on what 
data to collect. Second, data collection processes have to be developed for all 
stages of the study. Third, the study must be refined and detailed. Fourth, an 
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estimated budget needs to be developed. Finally, the study must be presented to the 
entire campus for review and comment. 

The tasks before the committee, therefore, are numerous and complex; the road 
ahead is littered with obstacles 3ut we have begun. Furthermore, the work of the 
committee has powerful promise. It holds out the potential for dramatically 
affecting the quality of the undergraduate experience at Eastern New Mexico 
University. 
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